In two dimensions, given an ordered set of three or more connected vertices (points) (such as in
connect-the-dots) which forms a
simple polygon, the orientation of the resulting
polygon is directly related to the
sign of the angle at any
vertex of the
convex hull of the polygon, for example, of the angle ABC in the picture. In computations, the sign of the smaller angle formed by a pair of vectors is typically determined by the sign of the
cross product of the vectors. The latter one may be calculated as the sign of the
determinant of their orientation matrix. In the particular case when the two vectors are defined by two
line segments with common endpoint, such as the sides BA and BC of the angle ABC in our example, the orientation matrix may be defined as follows: :\mathbf{O} = \begin{bmatrix} 1 & x_A & y_A \\ 1 & x_B & y_B \\ 1 & x_C & y_C \end{bmatrix}. A formula for its determinant may be obtained, e.g., using the method of
cofactor expansion: : \begin{align} \det(O) &= 1 \begin{vmatrix} x_B & y_B \\ x_C & y_C\end{vmatrix} -1 \begin{vmatrix} x_A & y_A \\ x_C & y_C \end{vmatrix} +1 \begin{vmatrix} x_A & y_A \\ x_B & y_B \end{vmatrix} \\[4pt] &= x_B y_C - y_B x_C - x_A y_C + y_A x_C + x_A y_B - y_A x_B \\[4pt] &= (x_B y_C + x_A y_B + y_A x_C) - (y_A x_B + y_B x_C + x_A y_C). \end{align} If the determinant is negative, then the polygon is oriented clockwise. If the determinant is positive, the polygon is oriented counterclockwise. The determinant is non-zero if points A, B, and C are non-
collinear. In the above example, with points ordered A, B, C, etc., the determinant is negative, and therefore the polygon is clockwise.
Practical considerations In practical applications, the following considerations are commonly taken into account. One does not need to construct the convex hull of a polygon to find a suitable vertex. A common choice is the vertex of the polygon with the smallest X-coordinate. If there are several of them, the one with the smallest Y-coordinate is picked. It is guaranteed to be a vertex of the convex hull of the polygon. Alternatively, the vertex with the smallest Y-coordinate among the ones with the largest X-coordinates or the vertex with the smallest X-coordinate among the ones with the largest Y-coordinates (or any other of 8 "smallest, largest" X/Y combinations) will do as well. Once a vertex of the convex hull is chosen, one can then apply the formula using the previous and next vertices, even if those are not on the convex hull, as there can be no local concavity on this vertex. If the orientation of a
convex polygon is sought, then, of course, any vertex may be picked. For numerical reasons, the following equivalent formula for the determinant is commonly used: : \det(O) = (x_B-x_A)(y_C-y_A)-(x_C-x_A)(y_B-y_A) The latter formula has four multiplications less. What is more important in computer computations involved in most practical applications, such as
computer graphics or
CAD, the absolute values of the multipliers are usually smaller (e.g., when A, B, C are within the same
quadrant), thus giving a smaller
numerical error or, in the extreme cases, avoiding the
arithmetic overflow. When it is not known in advance that the sequence of points defines a simple polygon, the following things must be kept in mind. For a
self-intersecting polygon (
complex polygon) (or for any self-intersecting curve) there is no natural notion of the "interior", hence the orientation is not defined. At the same time, in
geometry and
computer graphics there are a number of concepts to replace the notion of the "interior" for closed non-simple curves; see, e.g., "
flood fill" and "
winding number". In "mild" cases of self-intersection, with
degenerate vertices when three consecutive points are allowed be on the same
straight line and form a zero-degree angle, the concept of "interior" still makes sense, but an extra care must be taken in selection of the tested angle. In the given example, imagine point A to lie on segment BC. In this situation the angle ABC and its determinant will be 0, hence useless. A solution is to test consecutive corners along the polygon (BCD, DEF,...) until a non-zero determinant is found (unless all points lie on the same
straight line). (Notice that the points C, D, E are on the same line and form a 180-degree angle with zero determinant.) == Local concavity ==