This is a really interesting project, I had fun reading the post and will definitely check out the paper too!
One thing I've noticed in this (and similar low power chips) is that they are able to run at substantially lower frequencies - this paper mentions ~60 MHz and ~80 MHz for frontend and backend - which would be significantly lower than the operating frequency of ARM/Nvidia alternatives.
Is there any cost to running at such low frequencies? I didn't fully understand the FPS metric, but is there a minimum FPS that must be met for the chip to be considered usable? And is there anything else impacted by the frequency?
Thanks! The keyframe rate is the rate at which the chip will output a state estimate. Since this is for a real-time robotics use, it needs to be strongly related to real-time.
For example, a ~20 Hz output rate (you get estimates of robot position at 20 Hz) is good enough to then put a position stabilization controller around it, to hover the flying robot. 1 Hz would be too slow for that, the position control loop would be unstable. Likewise, outputing at 200 Hz probably has limited benefits since the physical drone's dynamics don't allow it to move fast enough for those higher rates to matter.
I assume the clock frequency is the minimum required so that the required numerical operations complete within the time budget. Using the 20 Hz example again for a crude illustration, all calculations need to complete within 50 ms (when a new calculation will start). So, if 1,000,000 clock cycles are required to complete the calculations (and for this project, this is very repeatable), then the clock period should be < 50 ns.
When I read the Navion paper, it was not clear to me what the algorithm constraints applied to fit the silicon do to the applicability to the VIO state space. For example, does the Navion implementation support fast flying UAVs, or precision drones that need to interact with moving objects. Do you have any insight in where the Navion solution sits in the larger VIO requirements state space?
Most of the choices made by the Navion team seem to be fairly generic. The image resolution is reasonable, could be higher but also quite acceptable (480p), image frame rate up to 171 fps and IMU up to 52 kHz are more than high enough. For the single Gauss-Newton step, it's tough to know for sure, but the paper presents a third-party dataset result, so probably not a very narrowing assumption. The linear solver logic shouldn't be affected by application -- the sparsity pattern is fixed based on the dynamics. In short, I think speed doesn't affect things too much.
The one that is most difficult to tell is the effectiveness of the image compression (they went from 8bits/pixel -> 1.625 bits/pixel), but the ablations were probably only done with the EuRoC dataset. I think the logic of that 4x4 compression block will be affected by the dynamic range in the image... so if the image has unexpectedly low contrast or high contrast it might have worse results, and this is environment-dependent. However, the only thing that affects is the SRAM for the frame buffers. Worst case, those may need to be upsized to hold a less-compressed image, accounting for the camera resolution.
This is a really interesting project, I had fun reading the post and will definitely check out the paper too!
One thing I've noticed in this (and similar low power chips) is that they are able to run at substantially lower frequencies - this paper mentions ~60 MHz and ~80 MHz for frontend and backend - which would be significantly lower than the operating frequency of ARM/Nvidia alternatives.
Is there any cost to running at such low frequencies? I didn't fully understand the FPS metric, but is there a minimum FPS that must be met for the chip to be considered usable? And is there anything else impacted by the frequency?
Thanks! The keyframe rate is the rate at which the chip will output a state estimate. Since this is for a real-time robotics use, it needs to be strongly related to real-time.
For example, a ~20 Hz output rate (you get estimates of robot position at 20 Hz) is good enough to then put a position stabilization controller around it, to hover the flying robot. 1 Hz would be too slow for that, the position control loop would be unstable. Likewise, outputing at 200 Hz probably has limited benefits since the physical drone's dynamics don't allow it to move fast enough for those higher rates to matter.
I assume the clock frequency is the minimum required so that the required numerical operations complete within the time budget. Using the 20 Hz example again for a crude illustration, all calculations need to complete within 50 ms (when a new calculation will start). So, if 1,000,000 clock cycles are required to complete the calculations (and for this project, this is very repeatable), then the clock period should be < 50 ns.
When I read the Navion paper, it was not clear to me what the algorithm constraints applied to fit the silicon do to the applicability to the VIO state space. For example, does the Navion implementation support fast flying UAVs, or precision drones that need to interact with moving objects. Do you have any insight in where the Navion solution sits in the larger VIO requirements state space?
Good question!
Most of the choices made by the Navion team seem to be fairly generic. The image resolution is reasonable, could be higher but also quite acceptable (480p), image frame rate up to 171 fps and IMU up to 52 kHz are more than high enough. For the single Gauss-Newton step, it's tough to know for sure, but the paper presents a third-party dataset result, so probably not a very narrowing assumption. The linear solver logic shouldn't be affected by application -- the sparsity pattern is fixed based on the dynamics. In short, I think speed doesn't affect things too much.
The one that is most difficult to tell is the effectiveness of the image compression (they went from 8bits/pixel -> 1.625 bits/pixel), but the ablations were probably only done with the EuRoC dataset. I think the logic of that 4x4 compression block will be affected by the dynamic range in the image... so if the image has unexpectedly low contrast or high contrast it might have worse results, and this is environment-dependent. However, the only thing that affects is the SRAM for the frame buffers. Worst case, those may need to be upsized to hold a less-compressed image, accounting for the camera resolution.