Gaming is a massive global community, and a similarly huge industry. One chunk of that industry, video gaming, is expected to be a roughly $140 billion global market in 2018, according to VentureBeat. And that figure doesn't even account for a form of gaming with a much longer history: tabletop games. And it's the intersection of those two -- video gaming and tabletop gaming -- that interests the California-based startup Spatial. They refer to themselves as a "mixed reality gaming company," and the reasons why became clear to me once I watched their demo videos:
"I think the gaming market is overflowing with massive anonymous content, and there are a lot of winners in that arena," says Spatial founder Kerry Shih. "But for me ... there’s sometimes a taxing and tiresome feeling to everything always being online, everything anonymous and massive. I think there’s a counter to that trend, towards a more interpersonal experience, and we hope to foster that in the form of tabletop gaming." The research bears out Shih's observations, as board games are projected to remain a growing global market in the coming years, and one that has embraced new tech. To take one example, tabletop classic Settlers of Catan is available in a variety of electronic formats, including iOS and Android apps, and one can play against opponents all over the world. "Tabletop has this unbelievably great venue where you’re shoulder to shoulder," Shih adds with a laugh, "so I can look at the anguish in my brother’s face as I beat him in a board game. That’s a really rich experience, with your family or friends or whomever." As someone who spent a good deal of time in his teens in hobby shops and convention centers playing in Magic: The Gathering tournaments, I can attest to this!
The aforementioned Settlers of Catan is part of the gaming experience that helped Shih and the Spatial team set their goals as they developed their product. "I think you could talk about our goal in terms of, I love Settlers of Catan, but I also love Clash Royale and Hearthstone (two popular online games which reference board gaming, collectible card gaming, and more). And with the first example you have piece play and the tactile feel, and the face to face contest gathered around the table, but with the second you have those amazing mechanics and graphics and the stuff I grew up on and love. And the whole project was borne out of, 'Why do I have to choose? Why can’t I have the awesome parts of digital gaming combined with tabletop piece play?'"
Where do you think Spatial fits into the next iteration of electronic gaming, I asked Shih. "I actually think it’s a bit of a trap to say 'next' as if there’s some curve where one form of gaming is better than the other," he points out. "But if you’re looking at the market as a whole, you have headset driven stuff like virtual reality (VR) and it’s kind of its own beast, it’s a very singleton experience and highly immersive. Headset-driven AR is another side of the spectrum where you’re wearing glasses or a hololens or something and it’s augmenting the digital imagery, as opposed to through the phone or tablet AR where you’re looking through a reconstituted world born from my camera and those images are morphed on the fly. And what we’re about is projected AR, which is using a display and the glass to render the images tethered to the piece play and the play space that you’re manipulating with your hands."
Watching Spatial's demo videos, I was reminded of the famous scene in Star Wars Episode IV: A New Hope, where Chewbacca and R2D2 play a sort of holographic chess game (the internet helpfully let me know that the game is called Dejarik, and supposedly it's pretty popular). So how did the company develop this space age tech? "One thing we’ve tried to solve is 'How do you build the world’s most inexpensive camera-driven IoT device?'” says Shih. "That’s something a lot of people are trying to solve, and we’re trying to crack it by using the camera for detection of objects in a gaming context. With the architecture, our idea was to let the hardware itself proxy the joystick controls and images from the camera, and then shovel it over to the mobile device and let that do the majority of the processing."
"One thing about computer vision and machine learning we found," he continues, "is that you don’t actually want HD images! Trying to find a 1 inch play piece in a tabletop size area, you don’t want a 1080p image because it would take forever to process, so what is the right resolution and encoding, and the tradeoff is between how much time it takes to encode the image on the sensor rig device and then unpack it on the mobile device."
Unsurprisingly for this emerging type of gaming tech, Spatial faced challenges in development. The team were creating a model in the gaming engine that could triangulate the two dimensional location of a play piece, detected by a camera, which then informs about a location on the table. This in turn communicates with the game engine rendering on a display that’s tilting backwards, bouncing off of glass and into the user's eyes. There were also the types of dollars and cents questions anyone who is selling hardware runs into: "We were working with what we could get off the shelf, parts online at low scale, tinkering around until we can get it working. We said 'we’re never going to be able to have an accessibly priced product if it needs any sort of Raspberry Pi or other chipset on there. Even a $10 computer would be too expensive, because once you mark it up for retail that’s the majority of your bill of materials." Finding the right components -- MCU, camera, and more -- was its own breakthrough, Shih says.
Shih has naturally been engrossed in the world of Spatial, but what developments in tech outside of his own company have caught his attention, I asked. "So having sort of heard the words machine learning, deep learning, neural nets, I sort of glossed over those when I originally read about them," he explains, "but now having used them in this project and seeing what other big companies are doing, it really is mindblowing. What it does to the programmatic model is remarkable; someone described it to me as: the old model was, ‘I write some rules, I give it some data, and it calculates an answer,’ and machine learning is ‘I have the data, I don’t know the rules, but here are the answers I expect.' And machine learning figures out the most optimal ruleset."
And finally, what's coming up next for Spatial? Says Shih: "Currently we’re all about the Kickstarter, we’re really happy with the response we’re getting from the campaign, and of course the market will tell us what they like and what they don’t like. Even in those three weeks of pre-campaigning we’ve come up with some game ideas from the feedback, so that’s awesome!"