Foreword:
Well, first and foremost, allow me to introduce myself.
My name is Vadim K.Riez, I’m in my forties, and when it comes to my occupations and hobbies, it turns out that I wear many hats: I am a Photography CTO, a Commercial photographer specializing in Technical and product photography, an Integration Ninja and a keen adept of everything based on innovative technology. I am absolutely in love with every aspect of photography, and it has been a major part of my life for the past twenty years. Over the years I have shot with many models and types of cameras, made by different manufacturers. But my personal preferences lay with Nikon. And this is mostly due to two reasons. First of all, starting from D800 – Nikon DSLR body abilities and functionality have met over 90% of my demands` checklist, and second of all – because I know exactly what to expect from the NEF files they produce. In my opinion, there is no such thing as a “perfect” camera, and all camera bodies from all manufacturers have their cons as well as “childhood diseases”. But when it comes to Nikon gear – I am very well aware of cameras` capabilities and shortcomings, and have found my ways to work around them. What drove me to write down this post – is ONE particular shortcoming of Nikon gear that drove me nuts up until the point I have decided to try and build a custom made solution. Ready for a bit of a long-read with technical whining?
Here we go!
The problem:
Over the past two years I have been working with the Z series cameras which superseded my D810 and D750 bodies. I am very happy with the integrability and flexibility that the Z bodies and the new Z mount provide. All of the limitations I have encountered in DSLR bodies were finally resolved in the Z series: The Live View is very detailed and has quite an impressive refresh rate, the shorter flange and bigger inner throat diameter of a new bayonet – allow for mounting these camera bodies pretty much everywhere with almost any kind of optics attached. Their size and weight allow for integration in all kinds of rigs, where mounting a DSLR camera would have call for more bulky solutions. Everything is really nice and I’m overly happy. Spare one thing: The lack of ability to remotely control the camera in a proper way. Let me explain: Apart from a simple remote control, that initiates the shutter through a dedicated camera port, up to this moment there has been no proper solution for doing more sophisticated things remotely (without touching the camera). For instance zooming in and out during Live View, or panning the zoomed area across the frame to check for sharpness or DOF distribution.
Well, to be completely honest – Nikon does offer a ML-L7 remote control that can be used (up to a certain degree) to control the functions of Z series. But it is not compatible (despite the latest firmware update) with Z6 and Z7 bodies. The new MC-N10 Control grip is still in development phase and up to this moment no compatibility chart has been published. Mobile Apps like Nikon SnapBridge is a dead end here as well. It can initiate remote Live View session, focus on a tapped point and take a picture, but that’s about it when it comes to its’ functionality. So the only means of doing something like zooming and panning remotely – is by using tethering software. But this approach has its own caveats that are not quite to my liking (more on this later). In comparison – Sony has implemented a simple and elegant solution to this problem in the form of Wireless Remote Commander unit that works flawlessly with all Alpha/Nex lineup, and enables the photographer to control different functions of Sony Cameras. We have to see if MC-N10 lives up to this functionality when it will be released. So, as a result of this continued frustration I was determined to find a solution to this problem.
The (long) quest for a solution:
First of all let me explain, why I am so eager to control the camera remotely. In my workflow, there are many shooting scenarios that demand me to use functions like Live View Zoom-IN and Zoom OUT, and panning the zoomed area across the frame, to check for sharpness/DOF distribution or to get a better look at REALLY small details. All of this, of course, calls for reaching the camera body in order to press the dedicated buttons. But what happens when the camera is out of reach? Simply, because it’s mounted on a C-Stand or a Boom that are extended to 3 meter (10 feet) height? Or being installed in a rig, or in an orientation that prevents me to reach the controls? Or when the camera body is being obstructed by flags or scrims? Or when I’m reaching a 5:1 Ratio in a microphotography session and every vibration of a camera body whenever I push the dedicated buttons – can throw me off the pinpointed focus plane position? Surely, at this point, an experienced photographer reading this post – would shrug and say “Bah! That’s why we have tethering software!” And in general terms I would agree. But the tethering approach bears many shortcomings as well.
One of my “simple” rigs, including an external HDMI monitor connected directly to camera HDMI port
Over the years I have worked with many tethering software solutions including but not limited to Nikon Camera Control pro 2, Capture One Pro, ControlMyNikon and Helicon Remote. And while they do provide a possibility to control the camera functions remotely – they all share a number of problematic issues. First of all, let me tell you that in my workflow – I do not need to shoot directly to a PC. I am perfectly fine with saving the NEFs to a memory card. Moreover – I seldom need to examine the file after it has been transferred to a PC. My workflow involves careful metering and adjusting the light ratios with the help of a flash-meter, so whenever it comes to examining the image on a larger screen – I do not it as an only possible way to determine exposure and lighting. When it comes to examining the DOF distribution, sharpness or composition examination – I have found that using an external HDMI monitor connected directly to a camera body does the job as well (Except for the times when you need to toggle overlays or to zoom in during playback. But that’s a completely different story).
Spoiler Alert! Here is a little fact not all of you are aware of: In Z series bodies, as well as Latest models of Nikon DSLRs (D780,D850) – It is possible to have the camera body attached to both, a USB connection for tethering and an HDMI connection for having a LiveView feed shown on an external monitor. Moreover, whenever the tethering software initiates a Remote LiveView session, the HDMI output retains full functionality and you can see the same image simultaneously in your tethering session as well as on an external HDMI monitor attached to the camera body. End of spoiler.
Same setup as above as presented on a camera screen. As can be deducted from the image – I make extensive use of Nikon Z cameras integration abilities, pairing them to View Cameras in order to use the Scheimpflug principle to control DOF distribution and achieve macro abilities beyond 1:1 scale
Anyhow, the issues that bug me most in various tethering software packages – are issues with Remote Live View modules of these programs. If you have worked once or twice with tethering solutions, you are already aware that in order to initiate functions other than changing parameters like IQ, Shutter Speed, Aperture and ISO values, or initiating a shutter button to take the shot – you have to toggle the Remote LV module on. And these modules, while initiating all the functions that I sought after – prove to be cumbersome and plagued with minor shortcomings regardless of software vendor.
So, what troubles me in Remote LV modules? A whole list of things:
First of all a terrible lag that is light-years behind “realtime”. Anyone who has tried digitally zooming during a remote LV session – can testify that the LV image on a PC screen lags significantly, and sometimes drops to a whopping 6fps, making fine manual focusing a painstaking process based on hit and miss success ratio. Second of all – poor representation of Live View “stream” resolution when it comes to rendering of fine detail. In many cases, especially when zooming in during remote live view, the programmers prefer to upscale the zoom region from an overall resolution, or to change the Live View resolution to a lower resolution in a way that the zoomed area will fill the entire screen while zoomed in. And as such it is really hard to see fine details in such a poor representation on a PC screen. Again, anyone who tried to pinpoint the focus plane using an enlarged (zoomed) mode in Remote LV session – can testify that it is time consuming and frustrating most of the time. Third of all – poorly implemented user interface. Not even one of the aforementioned tethering software packages – has key-bindings or shortcuts that allow the user to alter the Live view Zoom ratio or to move the zoomed area across the frame in an easy and convenient way. There is a possibility to use the mouse, and the mouse scrolling wheel of course, but this is much less intuitive than simply hitting a couple of buttons, like you do it on a camera body.
So, While I can address issues like real-time response of LV image representation and it’s quality by using an external HDMI monitor, connected to a camera body (HDMI has minimal lag compared to Remote LV by USB and always uses it’s native resolution (FHD/4K) regardless of LV zoom ratio), I still cannot control the camera this way. As I have already mentioned – I make an extensive use of camera’s HDMI interface, attaching the camera bodies I am working with to external monitors almost every time I shoot. But HDMI has only unidirectional connectivity. So while I can address the Image Quality and Lag issues – I needed to find a complimentary solution that would address the possibility to alter the zoom ratio and the zoomed area coordinates in a remote way.
All of this made me think. Can there be a way to communicate with the camera without actually making use of tethering software to control Remote LV functions? And if it is possible, could there be a way to control the camera by means of a really simple device with a USB interface? Perhaps a Netbook with an Atom processor or even a PC on a stick that can hang like a dongle from a USB cable connected to camera, without drawing too much power or having a bulky AC Adapter? This is when I have turned my gaze on a Nikon Official SDK Toolset. If you are not familiar with the term: SDK is a toolset in form of libraries that the Manufacturers provide to developers in order to aid them with writing software that could communicate with their (Manufacturers’) devices. These libraries are usually very well documented, so you get a detailed overview of all the functions defined in the toolset. So in case you have pondered: most of the tethering software that communicates with cameras is written with the help of different manufacturers` SDKs. So at this point – I thought to myself: “Let’s get the Nikon SDK for my camera bodies, and see whether I can find a software developer that can make use of this toolset and build me a little program that would be able to invoke all the needed functions remotely by means of USB. And if this software could omit the representation of Remote LV, and instead just fire away only the commands that are needed to invoke the necessary functions – that would be even greater!” I was so fascinated with this possibility, that I have immediately drawn a simple chart that lists such a setup.
And eventually my crazy idea worked!
The Implementation (Long and technical, but still worth a read):
I have then applied for downloading the Nikon SDK through the official link
I have downloaded the SDKs for my camera bodies and have opened the ZIP files. But imagine my disappointment that despite the fact that provided Documentation was very detailed, all the code samples and necessary libraries were written in C++. I am, by no means, a programmer. On a daily basis I deal with photography, rather than coding. And the last time I have written code that consisted of more than twenty lines of instructions – was way over twenty years ago. Moreover C++ is a little bit frightening for inexperienced coders, because .NET and current iterations of programming languages like C#, Python or Rust are way more “noob” friendly when it comes to dealing with code. I was disappointed and angry. There was no way that I could rewrite the code or to make use of it for my purposes all by myself. I simply lacked the needed skills. So I have started to Google-up, seeking for alternative solutions or maybe related projects that could spill some light on possible ways to deal with this C++ nightmare.
And what do you know? Apparently I was in luck! Because back in 2012 a talented developer named Thomas Dideriksen, has apparently ported the original Nikon SDK functionality to a more user friendly wrapper for C#: The Nikon SDK C# Wrapper
Although the last “official” update to these libraries was made at the time when the D500 hit the market, and the Mirrorless Nikons seemed more like a dream, this option seemed much more viable to provide me with an immediate solution rather than a bulky C++ structures from Nikon. And hence the code is being licensed under Creative Commons Attribution 3.0 Unported License – I was really happy. Moreover, the project, hosted by a Sourceforge.net platform – has a good discussion board with many contributors, which is a great sign that shows us that the project is being used and modified by many people who, just like me, are seeking to build their customs solutions that would work with Nikon cameras.
I have downloaded the project files, and started digging in to get a better understanding the code structure. Fortunately, the wrapper includes very useful sample files that illustrate the simplicity of use, and the discussion board is full of good examples how to solve specific issues if they arise. So even with my limited coding skills, I was able to grasp the general concepts needed to alter the code to my demands. So, with guidance and help provided to me by various talented developers (Warren, I owe you) – I have slowly started rewriting the code. I will not go into extensive explanations about code structure here, as it is not an appropriate platform for this, so the technical details will be spared for the sake of more fluent text.
Of course the code needed to be altered in order to communicate with newer Z series cameras. It took me a while, but I managed to establish a communication with my Z6 body, getting a list of all supported functions. As things enveloped – during the next few weeks, and with help of original Nikon SDK documentation I have rewritten or altered some parts of the code so it would be fully compatible with my camera body, and was able to invoke more complex functions like zooming in and out during live view and, eventually, changing the coordinates of the focus area (panning the zoomed in area across the frame). Next I was able to toggle the Focus Peaking and Exposure Preview functions of the camera by means of my software. And the rest of the time went to polishing the code, handling bugs, crashes and exception errors. So, all and all, this little software proved to do exactly what I was hoping it would do.
Conclusions and Peculiar insights:
With fair amount of help – I was able to build and compile a lightweight x64 program that can run on a simple ULV computer with Atom Processor onboard in Win10 64bit environment. It does not consume much memory or CPU resources and is fit to run on a PC on a stick or an Intel computestick that can be hung like a dongle from the cameras` USB port or a USB cable, while being powered by Power Bank or a simple 5V 2A AC adapter like the one used to charge a cell phone. It is being controlled by a simple Bluetooth Numerical Pad, which can be moved pretty much everywhere on the set, providing you are within reach of Bluetooth transmission radius. The + and – keys control the Zoom ratio, the numeric 2,4,6,8 keys control the movement of the focusing (zoomed in) area across the frame, 5 – resets the coordinates of the area back to the center of the frame, 1,3,7,9 keys are being used to swiftly “jump” to image corners, 0 Initiates Remote LV, / and * toggles focus peaking and exposure preview, and Enter is used to fire the camera’s shutter and to take a picture. So in general turns – it is a dream fulfilled, or a Solution to a shortcoming. Depends how you look at it…
Software GUI with camera connected
One interesting insight I have made during my attempts to optimize the software for low powered PCs, is a deeper understanding of actual structure of mechanism that is being used behind the scenes to when Remote LV and its functions are initiated. It turns out that what actually happens is that upon initiation of Remote Live View – the camera opens up a “stream” that is being transmitted by USB to the computer. This “stream” consists of a series of JPEG files (frames) acquired from Camera’s live view as well as additional “header” information that accompanies each frame within the stream. But what is much more interesting is the part of the “header” that accompanies each frame within the stream. It turns out, that this particular header (So called “Display Information Data”) bears all the information concerning camera live view functions and states like: The exact coordinates of Contrast focusing point or area as shown or set in camera, The Live View Image Area Size, The Focusing Judgment result, The focus Tracking status, the rotation direction and many more data. Why it is so interesting? Because, in general terms, it provides us with the ability to separate the transmitted “stream”. In simpler words: We can separate the JPEG data and the “header” data from each given frame. And as such, instruct the computer to disregard the JPEG part of a stream, and inquire the camera only for “header” part, allowing us to get the needed data that has current AF frame (area) coordinates, without getting or presenting the “visual’ part of the stream. And this is exactly what was needed in order to make this program run even on underpowered computers…
Afterword:
Did I reach my goal? – Yes, I have developed a stable lightweight software solution with small memory footprint that does exactly what I wanted it to do, without the downsides of more resource hungry Tethering software packages. Moreover, it can run on a very small form factor PC without catching too much space or causing cable clutter.
I am I happy with this solution? – Indeed. Perhaps I didn’t invent the wheel here, but I have found a viable solution to my shooting scenarios that would save me time and efforts. Especially for controlling the camera directly from the set, without reaching out to camera body in order to push the buttons, or reaching for the mouse to invoke the needed functions via the GUI of tethering software.
Was this little project interesting? – It was fascinating! I have learned so much in such a short amount of time, and all this new information will most definitely help me in my integration skills and developing or building more complex rigs or solutions.
What features does the software provide? – Currently the software is able to invoke the following functions via GUI or by means of a remote controller (I have chosen a Bluetooth Microsoft Wireless NumPad to do so): Zooming In and Out while in LiveView, panning the current focusing area (or zoomed section of the image) across the screen in any direction, Moving the current focusing area (or zoomed section of the image) to a pre-set areas like the edges of the frame and the frame’s center, Auto-rotating (remapping) of controller bound keys according to a camera orientation, initiating a capture command and saving the file to a camera’s memory card, or to a memory card AND a destination folder on PC, Toggling Exposure simulation ON and OFF, Toggling Focus peaking ON and OFF, Altering the camera exposure values (ISO, Aperture, Shutter Speed) from software’s GUI and showing current battery charge status.
Can this program operate anything other than Z6 body? – At this point – only the Z6 / Z6II / Z7 / Z7II bodies are supported. Simply because I was goal focused on getting my main rig camera to work. With Alternation of different parts of code that deal with specific parameters or functions of particular bodies – It can be suited to work with DSLR Nikon bodies as well. But right now, I want to give it a little rest, hence after three weeks of intense coding and bug hunting – I am starting to perceive the world around me in lines of code.
Can this software run alongside fully functional tethering software in order to transfer the image to a PC and present them on a large screen after the shot has been transferred? – This in an interesting possibility. In general terms, this software was developed only for controlling certain camera functions in a more convenient way that suits my workflow. It wasn’t destined to replace the tethering software for means of immediate NEF decoding or for presenting the shot images. Moreover, it was designed to work on underpowered PCs without a screen. But having thought about this – I have included the possibility to save the files not only to a cameras’ memory card, but to a destination folder on a PC as well. This way the Tethering program can monitor this folder, and present the images as they are being saved by means of “Monitor folder” in Adobe Lightroom, or “Hot folder” in Capture One Pro. It is worth mentioning that in this scenario – the tethering modules of the tethering software should be turned off or not to be initiated at the same time with my program. Otherwise the programs might interfere with each other.
Can my software run on systems other than Windows x64? At this moment the software is capable to run only on Widows 10 x64 systems, due to the fact that the C# Wrapper libraries are windows based.
Can I share my program with the world? – I suppose I can. But only at the point where the code is neat and elegant and it has zero bugs or crashes. Please notice that not being a full time programmer or a software publishing house – I will not be able to provide a full time technical support for this product. Moreover, with the Nikon upcoming MC-N10 control grip – the need for this program might be deprecated soon. But if I will choose to do so – I would gladly share the software on a donation-ware model.
Thank you for reading and best of luck to all of you!
Vadim K.Riez
www.vkriez.com
If you have an interesting idea for a guest post, you can contact me here.
