1

We have a big mural on a big wall. It is requested that, when viewing this mural on your handheld device, like a smartphone's camera, image overlays should be placed at specific positions within that mural (that mural has left out parts and the respective cutouts should be displayed on top).

Now, I followed the ar.js tutorial on image tracking and it kind of works, but I have the feeling that this is almost solely designed for horizontal and small placements. Like putting a car on your desk. The objects I managed to place on top of the mural are impossible to position, even when you add an orientation changer or rotate the objects.

This is what I have so far, tested with different sizes, rotations, positions:

<!DOCTYPE html>
<html>
<head>
  <script src="https://cdn.jsdelivr.net/gh/aframevr/aframe@1c2407b26c61958baa93967b5412487cd94b290b/dist/aframe-master.min.js"></script>
  <script src="https://raw.githack.com/AR-js-org/AR.js/master/aframe/build/aframe-ar-nft.js"></script>
  <title></title>
</head>
<body style="margin : 0px; overflow: hidden;">
  <a-scene
    vr-mode-ui="enabled: false;"
    renderer="logarithmicDepthBuffer: true;"
    embedded
    arjs="trackingMethod: best; sourceType: webcam;debugUIEnabled: false;"
  >
    <a-nft
      type="nft"
      url="url"
      smooth="true"
      smoothCount="10"
      smoothTolerance=".01"
      smoothThreshold="5"
      size="1,2"
    >
      <a-plane color="#FFF" height="10" width="10" rotation="45 45 45" position="0 0 0"></a-plane>
    </a-nft>
    <a-entity camera></a-entity>
  </a-scene>
</body>
</html>

It would be interesting to know how the sizing and widths and heights really function alltogether (for instance, in the documentation it say size is the nft size in meters, but is that really important? What about the children then?)

So I wondered, do I even need AR? Actually, it would be enough to detect image A in that mural (i. e. camera stream) and place another image B on top of that (or replace it), respecting the perspective.

Improve this question

1 Answer 1

Reset to default
1

The below is based on my experience.

The idea of creating the AR environment is to mimic the real-world surroundings the best you can. It's never perfect because of the approximations but there are ways to help the algorithms. One of them is the size of the marker. When using something like a camera that captures the 2d images of the real world, extracting the X and Y coordinates is "simple", but the depth must be deducted from the camera movement and the relative change in the object's position on the 2d image. The marker size is a hint of how far that particular object should be so I would say that the size of the marker is indeed important - if you decide to specify that.

Take a look at the example below:

example

This is a great simplification but try to imagine these two images are potential candidates for the marker position. But with a specified size - let's say you set it smaller than the real object - the camera would settle with the closer one.

Solution?

As far as I know, you don't need to specify the size of the marker - that way everything is left for the AR app to calculate.

But you can also take measurements and enter the correct size for better tracking.

Also, just a side note, please correct me if I'm wrong. Usually in A-frame attributes are separated with white-space and not ,. That would mean the size would be size="1 2" and not size="1, 2". But don't take my word for it, this would need to be verified.

What about the children?

The a-nft entity is placed where the marker was detected. It behaves like every other element so its children would inherit its placement as a local space. That would mean every transformation done in the local space would be placed on top of the parent transformation. For example, in A-frame, the position="X Y Z" is performed in local space.

Regarding the overlapping the images

If you are working with a rectangular image, that you want to project on a rectangular wall, then I would say your idea is good enough. I think that the most straightforward way would be to detect the 4 corners of the wall and warp the image so the corners fit (Four Corner Image Warp). That would cover the perspective transformation if you just use rectangular elements. But still, you have to somehow detect the mural.

But you may also think in advance in case one day you would like to enhance the experience and add some depth or 3D to the scene, then you would need the AR.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

thanks very much for your thoughts! I managed to get a somewhat decent result by fiddling around a lot. The main problem really is that ar.js expects you to look at a floor/ ground, so all the rotation and positioning is really hard to wrap your head around. I will try some more and look at the warp solution. Thanks!

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.