Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. In this piece of work, we are interested in creating a 3D virtual tour for an apartment.
However, for this apartment, no 3D information is available — instead, the input for the algorithm is something that we can obtain easily, in this case, a 2D floorplan and a set of images that we shot in the apartment. From this information, we would create a 3D floorplan that is not only faithful to the real one in terms of geometry, but the photos with the correct viewpoints are also to be assigned to the correct walls.
In order to accomplish this, one has to overcome a series of challenging problems. For instance, we have to estimate the layout of each room and find the location of the camera in each of these images.
Also, to obtain high-quality solutions, the goal is to extract as much information from the inputs as possible. The authors recognized that the floorplans provide way more information than we take for granted.
For instance, beyond showing the geometric relation of the rooms, it can also be used to find out the aspect ratios of the floor for each room. The window-to-wall ratios can also be approximated and matched between the photos and the floorplan.
This additional information is super useful when trying to find out which room is to be assigned to which part of the 3D floorplan. Beyond just looking at the photos, we also have access to a large swath of learning algorithms that can reliably classify whether we're looking at a bathroom or a living room.
There are even more constraints to adhere to in order to aggressively reduce the number of physical configurations, make sure to have a look at the paper for details, there are lots of cool tricks described there. As always, there is a link to it in the video description.
For instance, since the space of possible solutions is still too vast, a branch and bound type algorithm is proposed to further decimate the number of potential solutions to evaluate. And as you can see here, the comparisons against ground truth floorplans reveal that these solutions are indeed quite faithful to reality.
The authors also kindly provided a dataset with more than 200 full apartments with well over a thousand photos and annotations for future use in followup research works. Creating such a dataset and publishing it is incredibly laborious, and could easily be a paper on its own, and here, we also get an excellent solution for this problem as well.
In a separate work, the authors also published a different version of this problem formulation that reconstructs the exteriors of buildings in a similar manner. There is so much to explore, the links are available in the video description, make sure to have a look!
