Picture a scene that happens in many projects. The device resets once a day in the field. The software team checks the code and says: "The code is fine, it's a hardware problem." The hardware team measures the board and says: "The board is fine, it's a software bug." The project manager schedules a meeting. Two weeks later it turns out the problem is both: a short voltage dip during radio transmission that the code doesn't handle.
Nobody did bad work. The bug simply sat exactly on the line between the two teams, and neither of them saw the whole picture.
Where the hard bugs really live
Bugs that are entirely inside the code, or entirely inside the circuit, are found relatively quickly. The bugs that take weeks are almost always boundary bugs:
- Noise that looks like a bug. Sensor readings jump. Is it the code, or a trace running next to a switching source?
- A bug that looks like a faulty part. Communication fails. Is the component bad, or does the SPI configuration in the code not match it?
- Timing that works on the bench and breaks in the field. A longer cable, a different temperature, and suddenly messages get lost.
- Power consumption. The battery lasts two days instead of a month. Is it the hardware, or firmware that never puts the processor to sleep?
To solve a bug like that you need to know, at the same time, what is happening on the pin, in the timer, in the DMA channel and on the oscilloscope. When that knowledge is split between two people, every step of the diagnosis needs coordination.
Two examples from the field
An IMU sensor unit for a defense client. During board bring-up the sensor didn't respond at all. From the code side there were two problems that looked exactly the same: no reply from the sensor. In reality one was a wrong default SPI configuration, in the code. The other was a connector wired backwards, in the hardware, because a datasheet drawing showed the part from below. A software engineer alone would have searched the code. A hardware engineer alone would have measured the connector. The engineer who did both found both. The full project.
A control board with three high-speed RS422 channels. Communication was corrupted, and the first suspect was the code. The problem was in the cabling: a transmit line split to two connectors with no termination. Someone who knows both the protocol and the physics of the line spots that fast. The full project.
What else you gain when one engineer owns everything
- Better design decisions. When the person designing the board knows what the code will need, they choose the right pin for the timer, leave a DMA channel free and put test points in the right places. Many problems are never created.
- No interface document that goes stale. Between two teams you need a document describing every pin and register, and someone has to update it after every change. When one person does both, the interface simply works.
- One responsibility. When something doesn't work there is one address. No "that's the other team's problem".
- Fewer meetings, more work. Time that would go to coordination goes to development, and you don't pay for it.
When it fits less
To be fair: a very large system, with dozens of boards and hundreds of thousands of lines of code, needs a team, and one person cannot do everything. Even then, each subsystem should have one engineer who understands both its hardware and its code, and someone should see the full picture.
For most electronic products, such as controllers, portable medical devices, IoT units, control boards and test systems, one engineer who owns the hardware and the firmware is the fastest and cheapest way to a product that works.
What to ask a development company before you start
- Who will design the board, and who will write the code? Is it the same person?
- When there is a problem integrating hardware and software, who is responsible for solving it?
- Will I talk directly to the engineer, or through a project manager?
At ProtoMakers the answer to all three questions is the same engineer. How we work.
Written by Roman Alperovich, founder and lead engineer at ProtoMakers.