Say you want to calculate overlap between an object and a rectangular grid of cells that may be solid or not solid. Assume you know the position of an object's center (in world space, relative to the grid's origin), its radius (distance from the center to the edge of the bounding box) in X and Y directions. Also assume at first that the radius is less than half the length of a grid square, that is, the object is no larger than a grid cell.
This algorithm detects whether the block is penetrating a solid cell and which way it should be pushed so that it is no longer inside a solid cell. It has been prototyped in Python as part of Wrecking Ball Boy.
Numerous two-dimensional games for the NES, Super NES, and other 2D platforms make their maps out of a rectangular grid of tiles. Each cell in the grid can be solid or empty. A solid cell, for example, is one that an object is never supposed to enter, such as a wall or floor. Some cells may be directionally solid, in that they are solid if an object enters from one direction but allow passing through in the opposite direction, such as most platforms in World 1-1 of Super Mario Bros. 2. Some cells may be solid in one part and empty in another part, such as slopes.
A common method of detecting collisions involved sampling the grid at various points relative to a game object's position. A widely cited article about the implementation of a platform game on a 1980s 8-bit machine, "Programming M.C. Kids", discusses how the game detects collisions by sampling the grid at eight points around an object's perimeter, with some points not checked depending on the direction of motion. And once the program detects a collision, it must respond to the collision by ejecting the object and possibly changing its velocity or movement state.
Several games are known for glitchy collision detection and response in edge cases. Super Mario Bros. briefly puts Mario into the standing state when he enters a wall, allowing for wall jumping glitches, and the popular method of entering its World 36-1 relies on a flaw in its wall ejection. Tool-assisted speedruns of Mega Man series are known for exploiting bugs in the wall ejection behavior. More than once, I've got Sonic stuck inside a ramp in Sonic the Hedgehog 2 for Sega Genesis. And some games move objects around within the background, meaning one cell changes from solid to empty while another changes from empty to solid, which may confuse some of the game logic. I guess some of the poor collision response in some third- and fourth-generation games is due to a lack of rigorous reasoning about where to sample and what to do in each case.
This lack of rigor is what I aim to correct with the following algorithm that detects whether an object overlaps a solid cell and, if so, the closest position that does not overlap a solid cell.
This step characterizes the contour of the tiles that make up the neighborhood around the object.
The first thing to do is round the object's position to the nearest intersection of grid cells and calculate the displacement (dx, dy) from the intersection to the object's center.
Then sample the solidity of the cells at the four corners of the object's bounding box. Sampling the solidity involves rounding each corner to the nearest cell center and retrieving the tile there.
On some retro platforms, it may be faster to round the position once and then retrieve other neighboring cells relative to the first cell's position. This is also true if the object's bounding box is entirely embedded in a solid cell, as an empty tile may be nearby toward which it can be pushed. In this case, you can retrieve the solidity of the four cells surrounding that intersection. Then if the object's center is not currently embedded in a solid cell, and the absolute value of a displacement coordinate is greater than or equal to the radius, then the bounding box is completely in one of the squares, not straddling a grid line. Copy the column to the other column if dy <= -r or dy >= r, or copy the row to the other row if dx <= -r or dx >= r. After this, only those edges that the bounding box straddles will be included in the contour.
To handle directionally solid tiles, the engine must store the previous position of the object accurate to at least one tile. Assume without loss of generality that a particular tile type cannot be passed through downward. Treat such a tile as solid only if it's on the bottom row of the neighborhood and the previous position was completely above that tile.
Assign each solid cell a place value: 1 for top left, 2 for top right, 4 for bottom left, and 8 for bottom right, and combine these using bitwise OR (or equivalently addition). This associates each of 16 possible neighborhoods with a hexadecimal nibble:
In this representation, copying a row or column can be done with an AND, a shift of one of two bits, and an OR.
At this point, there are six general contour shapes, corresponding to the six distinct blocks in the video game Lumines or the six cases of the marching squares algorithm that are distinct up to rotation.
Some contour shapes require first reducing the number of cases by expanding any isolated solid cells to the full width of the whole block.
The object is colliding with the corner of a solid tile. First determine whether the corner is actually within the object. For a circular object, evaluate the inequality dx2 + dy2 < r2, either through multiplication or through lookup tables of y = x2. If not, reject the collision. Otherwise, use dx and dy to determine how far to try pushing the object out of the corner.
Extension to slopes and to objects larger than a cell is left as an exercise for the reader. For large objects, it involves sampling the object's perimeter. This may fail if a solid tile is completely embedded in an object.