Do I have to install a transponder in my amateur-built aircraft?
In general, no, a transponder is not required equipment. However, if you wish to operate in class A, B, or C airspace, or at an altitude of over 10,000' MSL, or within a 30 nautical mile radius of the primary airport in class B airspace, you will need a transponder and altitude encoder (commonly referred to as "mode C"). The regulations regarding transponder and altitude-reporting requirements are found in 14 CFR 91.215.
There is an exception to the regulations as they apply to the 30 nm Mode C "veil" around the Class B airports. This exception is found in § 91.215(b)(3), which states that if the aircraft is a glider or balloon, or was not certificated with an engine-driven electrical system, it can be operated within the "veil" without a Mode C transponder. This is allowed so long as these operations are below the upper limit of the Class B airspace (or 10,000 MSL, whichever is lower) and not actually within Class A, B, or C airspace.
The key to this exception is whether or not your aircraft has an "engine-driven electrical system". A good way to look at this is, if your aircraft does not have generator or alternator that is turned by the engine, and that recharges a battery in flight, you probably do not have an "engine driven electrical system".