In another blog post I talked about the design behind the UI created however this blog post will more cover the implementation and code behind the UI including interaction with it as well as interaction with the environment and hand tracking.
Hand Tracking And Activation
The handing tracking code is one of the most important aspects of the UI as it gives it the AR style control and interaction by following the hand. The code for this went through a few different versions with it changing based upon our own testing and feedback.
The initial implementation was fairly tricky as I ran into a number of issues, more with regards to making the panel follow the player’s hand smoothly. To make this blog post easier to read, the main class is split up into different sections but a link to the final version will be provided at the end.
When first writing the UpdatePosition method, the panel object was originally set as a child of the palm in the hand with the local position set with a positive y to hover above it. Unfortunately this caused the panel to jitter a lot due to the hand constantly shifting around and causing miniscule movement in the hand to appear amplified with the panel movement. As such, the function was rewritten to instead constantly update the position of the panel in relation to the hand which allowed me to apply a smoothing effect to it.
In the new function, the palm position is fetched then an offset is added so it appears above the hand. This is done by multiplying the rotation of the palm with the offset in the general direction and adding it to the position. Linear Interpolation (Lerp) is then used to move the panel from it’s current position to the new position with the time parameter being the absolute value (in this case so it is not negative) of the two square magnitudes of the positions taken away from each other. This is a little unusual compared to most smoothing algorithms however it worked well in this case due to the small movements it will be making and means that the further apart the two positions are, the faster it will move.
For activating the panel when the hand interacts with the wall, the interactable has it’s own script to detect if it’s a hand and if it should activate the panel. That script will be covered in another blog post however that script calls the Attach method on the panel which causes the panel to set the hand it’ll follow as well as load the log data from the interactable.
It is fairly simple self explanatory so I won’t go into a lot of detail but it resets any hand tracking data it keeps then updates the relevant child components with the required text as well as play any audio that is part of the log. The only unusual part is the call to DelayedUpdate in DynamicScroll but this is covered later in the blog post.
The hand retracking is fairly simple code and relies upon some expected Leap behaviour. If the hand it was attached to is no longer visible, the hand object is destroyed which causing the hand variable to be set to null. If that is the case, it then attempts to reattach to the correct hand by looping through the current active hands that the controller is tracking, seeing if it matches the criteria (for now, if it is the right hand) then setting hand variable again and resetting any tracking data. It will keep attempting to retrack the hand for retrackTime and if it fails to, it deactivates the canvas. UpdatePosition is only ever called if the hand variable is not null otherwise the Update function terminates before that.
The full class can be found here.
Two new features were added in the newest version so that the UI is more functional; These features are the audio button and log buttons. Some changes were also made to existing features such as the offset for the hand to be on the left as well as changing the UI to disappear on hand tracking loss then reappear once refound.
The audio code is very simple with most of the heavy lifting being done in the AudioQueue class. For this code, the AudioToggle is called when the audio button is pushed with the state of the button being passed in. The Audio function is called every Update and is mostly to update the look of the audio button automatically.
The code can be found here.
Sorting Log Entries
The log entries are sorted by length of the message currently. In the future, an option will be added to the UI to allow sorting via different categories (such as time it was created, alphabetical etc) however this is currently not implemented.
A custom class was created that implements the interface IComparer then an instance of it stored. This class implements the Compare method which compares two objects together, in this case two different LogData objects. If the value returned in less than 0, that means the first parameter is less than the second one and should be “behind” it in the list. The opposite is true for it is greater than 0 and if 0 is returned then they are equal.
In C#’s implementation of Sort, it does not provide a stable sort which means that if elements are equal, they may not remain in the same order. It also uses different algorithms depending on the number of elements in the list as different algorithms scale differently depending on the number of elements. In this case, as it’s most likely there will be less than 16 elements in the list then it uses the insertion sort algorithm.
On average, using Sort is O(n log n) however at worst it can be O(n^2) due to the different algorithms it may use. As it will most likely be insertion sort then the average performance is O(n^2) and best performance is O(n).
However due to the simple nature of the Comparer, it could be simplified to a lambda expression instead. This does the same as the Comparer however inline in the method call instead. The first two variables in the brackets are the parameters with the statement to the right of the => being what is returned. Using a lambda instead of a class saves on a small amount of memory as well as making the code cleaner and easier to read without the need to look at another class.
The audio button is based upon the default leap motion toggle button however with some custom changes. These changes allow the button to exist as part of a Canvas instead of being a 3D object as well as firing custom events on state change instead of relying on overriding method like the default implementation.
Most of the heavy lifting is done in the ButtonToggleBase which is part of the Leap Motion SDK and it handles the physics and interaction of the button with the hand. However all this does is cause the button to change colour so I expanded upon this class to create additional features as well make it expandable and reusable for future use via adding events and making it work as part of a Canvas.
Firstly I created a new UnityEvent by extending the class and passing in a bool as the generic type. The type passed in defines what the parameter of the event is as this is for a toggle button, the use of bool is fine to represent off and on as false and true. The Serializable attribute was added so Unity knew to save it with the scene then a variable created in the ButtonToggle to store this event. By creating this variable and exposing it to Unity, this allows the Unity Editor to then display an events section similar to how a Button would display.
I can then tell it to call the AudioToggle method on the HandInfoPanel script attached to the Hand Mount. As the parameters for the AudioToggle method matches the type for the event, this is known as a dynamic event which means that the value of the event is passed in as a parameter.
To trigger the event to fire, I override the relevant methods from ButtonToggleBase then call Invoke with what the value of the event should be.
The method SetToggleState allows the forcing of setting the current state of the button which is useful for being called by external scripts. Two of its parameters have default values which means that when the method is called, those two arguments can be left out and instead the default values will be used. The script calling the method can still however pass in these arguments and override the default values.
Implementing the scrolling text was the most tricky part to get working correctly. Although the Leap Motion SDK does provide this feature, it runs into the same issues as the button in that it exists as a 3D object and that it does not allow for dynamically resized text. This is an issue as the log entries will change in length and the panel will need to adjust for that so to fix this, I overrode the default implementation of the scrolling text and adding the functionality for changing text content.
All of the scroll handling is still managed in the base ScrollBase class provided with the SDK and instead this script instead updates the bounds of the text that the script uses for its calculations. Two new components were also added to the GameObject that displayed the text which allows it to resize itself as required. This is the Content Size Fitter and Layout Element components. The Content Size Fitter allows for it to resize with the Layout Element setting certain parameters on how it can resize.
For the script itself, the main part of it is the ContentBoundUpdate which changes the position of the bounds and position of the content. In this case, the content is the GameObject of the text for the panel. It firstly calculates the height midpoint of the transform which it then uses to position the top and bottom bounds. It can do it this way as it’s setting the local position (so the position relative to the parent) and the bounds are the child of the content object. It then sets the position of the content to be at the top most point and updates the Layout Element with its own dimensions so the Content Size Fitter can be more dynamic.
However this does have some issues of it’s own due to the way Content Size Fitter. The fitter does not run its calculations until the next frame update meaning if you change the text in one frame, on that same frame the dimensions of the object will be incorrect. To work around this, the text is set then ContentBoundUpdate is called 0.1 seconds later after the fitter has run its calculations and adjusted accordingly.
Overall I ran into many issues with getting the UI to work as intended however I overcame most of these issues. To do this, I customised and overrode some default methods of Leap Motion widgets to get the behaviour needed for the UI to function correctly within the context of the story such as allowing them to work as part of a Canvas and being more dynamic with events and resizing.
All the elements of UI are functional although some more work could be done, mostly on the design side of things.