embed_lane.h

Source: include/embed_lane.h

The lane beside the machine word: the widest unit that the target moves and computes in one operation. Also the operations that each bulk pass over memory runs in.

Note: The word, embed_word, is the register, and each count, mask and carry runs in it. The lane is the widest vector register of the target, EMBED_LANE_BITS wide. This header derives its width, as embed_types.h derives the width of the word. It is 512 bits under AVX-512, and 256 under AVX2 and on a CUDA device. It is 128 under SSE2 or NEON, and the width of the word where the target has no vector unit.

Note: A bulk operation steps one lane at a time on each target: a copy, a fill, a compare, a blend of pixels. At compile time, the lane takes the width that the target has.

Note: There are two arms, and each defines every operation. The vector arm is the vector type of the compiler: EMBED_LANE_U32S lanes of 32 bits in its widest registers. The plain operators run over it, and the compiler lowers it to the instructions of the target. The portable arm is a struct of the same 32-bit lanes, with one statement for each word in each operation. A CUDA thread holds that struct in registers.

Note: The 32-bit lanes are the unit of the arithmetic. A pixel is one, and the product of two bytes fits in one. The byte masks tell where bytes are equal or zero, with one bit for each byte. The register holds the mask whole, as one integer.

Note: This header includes no header of the C library. A load or a store at any address goes through the raw view of the lane, as for the word.

Note: Each entry has EMBED_HOST_DEVICE. Under nvcc one definition compiles into the host pass and the device pass. The device pass takes the portable arm at 256 bits. So a kernel calls the same functions that a host walk calls.

Warning: Where the build sets EMBED_LANE_PORTABLE to 1, the header takes the portable arm on any target. This header defines both values of each arm flag. Test them with #if.

Types

embed_lane

#if EMBED_LANE_VECTOR
typedef embed_u32 embed_lane EMBED_VECTOR(EMBED_LANE_BYTES);
#else
typedef struct
{
    EMBED_ALIGN(EMBED_LANE_BYTES) embed_u32 at[EMBED_LANE_U32S]; /**< The 32-bit lanes, the lowest address first. */
} embed_lane;
#endif

The lane, EMBED_LANE_BYTES wide. On the vector arm it is EMBED_LANE_U32S 32-bit lanes in the vector type of the compiler. On the portable arm it is a struct with the alignment of the lane.

Note: On both arms lanes in a row lay out as the registers do, each lane at a multiple of its width.

• at: The 32-bit lanes, the lowest address first.

embed_lane_bytes

#if EMBED_LANE_VECTOR
typedef embed_u8 embed_lane_bytes EMBED_VECTOR(EMBED_LANE_BYTES);
#endif

The lane seen as its bytes, which the byte masks compare in.

embed_lane_chars

#if EMBED_LANE_VECTOR
typedef char embed_lane_chars EMBED_VECTOR(EMBED_LANE_BYTES);
#endif

The lane seen as char, the shape the compiler's byte mask builtin reads.

embed_lane_raw

#if EMBED_LANE_VECTOR
typedef embed_lane embed_lane_raw EMBED_RAW;
#endif

The lane again, readable and writable at an address that is not a multiple of its width, as embed_raw_word is the word.

Note: EMBED_RAW lowers the alignment to one and permits access through an alias. A load or a store through this type has a defined result at any address. It is one instruction.

Note: The vector arm alone has this type. A type attribute on a typedef of a struct has no effect after the definition of the struct. The alignment of a struct cannot go below the alignment of its members. So the portable arm reads and writes a lane one embed_raw_word at a time.

embed_lane_mask

#if EMBED_LANE_BYTES <= 32u
typedef embed_u32 embed_lane_mask;
#else
typedef embed_u64 embed_lane_mask;
#endif

The byte mask that the byte operations of a lane give, one bit for each byte, the lowest byte at the lowest bit. It is 32 bits wide up to a 32-byte lane, and 64 above it.

embed_lane_words

#if EMBED_LANE_PLAIN
typedef union
{
    embed_lane lane;                                       /**< The lane. */
    embed_word words[EMBED_LANE_BYTES / sizeof(embed_word)]; /**< The same bytes as words. */
} embed_lane_words;
EMBED_HOST_DEVICE EMBED_INLINE embed_lane embed_lane_load(const void *at);
#else
EMBED_HOST_DEVICE EMBED_INLINE embed_lane embed_lane_load(const void *at);
#endif

A lane loaded from any address.

• at: The address of the EMBED_LANE_BYTES bytes to read, at any alignment.

**Returns:** The bytes as a lane.

• lane: The lane.
• words: The same bytes as words.

Macros

EMBED_LANE_BITS

#ifndef EMBED_LANE_BITS
#if defined(__CUDA_ARCH__)
#define EMBED_LANE_BITS 256
#elif EMBED_VECTOR_REGISTER_BITS != 0
#define EMBED_LANE_BITS EMBED_VECTOR_REGISTER_BITS
#elif EMBED_WORD_BITS > 32
#define EMBED_LANE_BITS EMBED_WORD_BITS
#else
#define EMBED_LANE_BITS 32
#endif
#endif

The lane's width in bits, derived from the target's widest vector register.

Note: EMBED_VECTOR_REGISTER_BITS comes from the compiler's own instruction set macros. A CUDA device takes 256, a thread's eight 32-bit registers. A target with no vector unit takes the word's width, and 32 where the word is narrower, since a lane holds whole 32-bit lanes.

Note: A build that wants another width defines this macro ahead of this header, as for EMBED_WORD_BITS. The assertions below hold any width to whole 32-bit lanes, whole words, and at most 512 bits, the width that the byte mask can hold.

EMBED_LANE_BITS_WAS_OVERRIDDEN

#ifndef EMBED_LANE_BITS
#define EMBED_LANE_BITS_WAS_OVERRIDDEN 0
#else
#define EMBED_LANE_BITS_WAS_OVERRIDDEN 1
#endif

Set to 0 where this header derives the lane width, 1 where the build defines it.

Note: Both arms define this macro. Test it with #if.

Set to 1 where the build defined EMBED_LANE_BITS ahead of this header.

EMBED_LANE_BYTES

#define EMBED_LANE_BYTES (EMBED_LANE_BITS / 8u)

The lane's width in bytes.

EMBED_LANE_U32S

#define EMBED_LANE_U32S (EMBED_LANE_BYTES / 4u)

The 32-bit lanes a lane holds.

EMBED_LANE_WORDS

#if EMBED_LANE_U32S == 1u
#define EMBED_LANE_WORDS(word_) word_(0)
#elif EMBED_LANE_U32S == 2u
#define EMBED_LANE_WORDS(word_) word_(0) word_(1)
#elif EMBED_LANE_U32S == 4u
#define EMBED_LANE_WORDS(word_) word_(0) word_(1) word_(2) word_(3)
#elif EMBED_LANE_U32S == 8u
#define EMBED_LANE_WORDS(word_) word_(0) word_(1) word_(2) word_(3) word_(4) word_(5) word_(6) word_(7)
#elif EMBED_LANE_U32S == 16u
#define EMBED_LANE_WORDS(word_) \
    word_(0) word_(1) word_(2) word_(3) word_(4) word_(5) word_(6) word_(7) word_(8) word_(9) word_(10) word_(11) \
        word_(12) word_(13) word_(14) word_(15)
#endif

word_ expanded one time for each 32-bit lane of the lane, with the place of that lane. So an operation of the portable arm is one statement for each word, with no counted walk.

• word_: A macro of one parameter, the word's place from 0.

EMBED_LANE_PORTABLE

#ifndef EMBED_LANE_PORTABLE
#define EMBED_LANE_PORTABLE 0
#endif

Set to 1 by the build to take the portable arm on any target, else 0.

Note: To check the portable arm on a host with the vector type, build with this macro at 1. Then the host runs the arm that the device takes.

EMBED_LANE_VECTOR

#if EMBED_HAS_VECTOR_SIZE && !EMBED_LANE_PORTABLE && !defined(__CUDA_ARCH__)
#define EMBED_LANE_VECTOR 1
#define EMBED_LANE_PLAIN 0
#else
#define EMBED_LANE_VECTOR 0
#define EMBED_LANE_PLAIN 1
#endif

The arm the lane takes: EMBED_LANE_VECTOR is 1 and EMBED_LANE_PLAIN 0, or the other way.

Note: The vector arm needs the vector type of the compiler, EMBED_HAS_VECTOR_SIZE. A CUDA device does not take it, because its compiler keeps the lanes of a thread in registers from the struct.

EMBED_LANE_MASK_ALL

#if EMBED_LANE_BYTES >= 64u
#define EMBED_LANE_MASK_ALL ((embed_lane_mask) ~(embed_u64)0)
#else
#define EMBED_LANE_MASK_ALL ((embed_lane_mask)(((embed_u64)1 << EMBED_LANE_BYTES) - 1u))
#endif

The mask with every byte of the lane set.

EMBED_LANE_NIBBLE_GATHER

#define EMBED_LANE_NIBBLE_GATHER 0x01020408u

The multiplier that gathers the four bytes of a 32-bit lane into one nibble. Each byte holds 1 or 0 in its lowest bit, and the lowest byte goes to the lowest bit.

Note: A multiply puts the four bits at bits 24 to 27 alone. The four powers of two in the multiplier set each bit of the product one time, with no carry between them. The top byte of the product is the nibble.

Functions

embed_lane_mask_first

EMBED_HOST_DEVICE EMBED_INLINE embed_u32 embed_lane_mask_first(embed_lane_mask mask);

The index of the first byte set in a mask.

• mask: A mask with at least one byte set.

**Returns:** The byte, counted from the first.

Note: The trailing zero count where the compiler has it, else the lowest bit isolated and placed by six masked steps.

embed_lane_load_aligned

EMBED_HOST_DEVICE EMBED_INLINE embed_lane embed_lane_load_aligned(const void *at);

A lane loaded from an address aligned to the lane.

• at: The address of the EMBED_LANE_BYTES bytes to read, a multiple of EMBED_LANE_BYTES.

**Returns:** The bytes as a lane.

Warning: C leaves a load from an address not aligned to the lane undefined here. embed_lane_load reads from any address.

embed_lane_put

#if EMBED_LANE_PLAIN
EMBED_HOST_DEVICE EMBED_INLINE void embed_lane_put(void *dst, embed_lane lane);
#else
EMBED_HOST_DEVICE EMBED_INLINE void embed_lane_put(void *dst, embed_lane lane);
#endif

A lane put at any address.

• dst: The address of the EMBED_LANE_BYTES bytes to write, at any alignment.
• lane: The bytes.

embed_lane_put_aligned

EMBED_HOST_DEVICE EMBED_INLINE void embed_lane_put_aligned(void *dst, embed_lane lane);

A lane put at an address aligned to the lane.

• dst: The address of the EMBED_LANE_BYTES bytes to write, a multiple of EMBED_LANE_BYTES.
• lane: The bytes.

Warning: C leaves a store to an address not aligned to the lane undefined here. embed_lane_put writes to any address.

embed_lane_fill_u32

EMBED_HOST_DEVICE EMBED_INLINE embed_lane embed_lane_fill_u32(embed_u32 val);

Every 32-bit lane of a lane set to val.

• val: The 32-bit value.

**Returns:** The lane.

Note: On the vector arm a scalar added to the zero vector is the scalar in every lane.

embed_lane_fill_byte

EMBED_HOST_DEVICE EMBED_INLINE embed_lane embed_lane_fill_byte(embed_u8 val);

Every byte of a lane set to val.

• val: The byte.

**Returns:** The lane.

embed_lane_and

EMBED_HOST_DEVICE EMBED_INLINE embed_lane embed_lane_and(embed_lane one, embed_lane other);

The bitwise and of two lanes.

• one: A lane.
• other: A lane.

**Returns:** The and.

embed_lane_or

EMBED_HOST_DEVICE EMBED_INLINE embed_lane embed_lane_or(embed_lane one, embed_lane other);

The bitwise or of two lanes.

• one: A lane.
• other: A lane.

**Returns:** The or.

embed_lane_xor

EMBED_HOST_DEVICE EMBED_INLINE embed_lane embed_lane_xor(embed_lane one, embed_lane other);

The bitwise exclusive or of two lanes.

• one: A lane.
• other: A lane.

**Returns:** The exclusive or.

embed_lane_add_u32

EMBED_HOST_DEVICE EMBED_INLINE embed_lane embed_lane_add_u32(embed_lane one, embed_lane other);

The 32-bit lanes of two lanes added, each with wrap around at 32 bits.

• one: A lane.
• other: A lane.

**Returns:** The sums.

embed_lane_sub_u32

EMBED_HOST_DEVICE EMBED_INLINE embed_lane embed_lane_sub_u32(embed_lane one, embed_lane other);

The 32-bit lanes of other taken from those of one, each with wrap around at 32 bits.

• one: A lane.
• other: A lane.

**Returns:** The differences.

embed_lane_mul_u32

EMBED_HOST_DEVICE EMBED_INLINE embed_lane embed_lane_mul_u32(embed_lane one, embed_lane other);

The 32-bit lanes of two lanes multiplied, the low 32 bits of each product kept.

• one: A lane.
• other: A lane.

**Returns:** The products' low halves.

embed_lane_shift_right_u32

EMBED_HOST_DEVICE EMBED_INLINE embed_lane embed_lane_shift_right_u32(embed_lane lane, embed_u32 count);

The 32-bit lanes of a lane shifted right by count bits, zeros shifted in.

• lane: A lane.
• count: The bits, below 32.

**Returns:** The shifted lanes.

embed_lane_shift_left_u32

EMBED_HOST_DEVICE EMBED_INLINE embed_lane embed_lane_shift_left_u32(embed_lane lane, embed_u32 count);

The 32-bit lanes of a lane shifted left by count bits, zeros shifted in.

• lane: A lane.
• count: The bits, below 32.

**Returns:** The shifted lanes.

embed_lane_bytes_equal

EMBED_HOST_DEVICE EMBED_INLINE embed_lane_mask embed_lane_bytes_equal(embed_lane one, embed_lane other);

Which bytes of two lanes are equal, one bit a byte, the lowest byte the lowest bit.

• one: A lane.
• other: A lane.

**Returns:** The mask.

Note: On the vector arm, the code compares the bytes as a byte vector, and each equal byte becomes all ones. Where the compiler has the byte mask builtin for the width of the lane, it reads the mask in one instruction. That builtin is EMBED_BYTE_MASK_256 or EMBED_BYTE_MASK_128. Otherwise one multiply over the full lane gathers the four bits of each 32-bit lane into a nibble, and the code reads out the nibbles. On the portable arm, each 32-bit lane finds its zero bytes by the carry test of the word, and gathers them the same way.

embed_lane_bytes_zero

EMBED_HOST_DEVICE EMBED_INLINE embed_lane_mask embed_lane_bytes_zero(embed_lane lane);

Which bytes of a lane are zero, one bit a byte, the lowest byte the lowest bit.

• lane: A lane.

**Returns:** The mask.