Bio-aggregate composites (BACs) are typically formed by binding plant origin aggregates using organic or inorganic binders. Composite boards are being manufactured from hemp shives and Portland cement or lime and such material is associated with the so-called “hempcrete”. To reach a low greenhouse gas emission rate, alternative binders must be considered. Gypsum binder releases a seven times lower amount of CO2 during production compared with Portland cement, while waste gypsum can be even more efficient. In this research, gypsum-based BACs were elaborated and tested. Phosphogypsum was evaluated as an alternative binder. The objective of the research was to evaluate the fire resistance of gypsum- and phosphogypsum-binder-based BAC. In this study, the amount of binder was varied and BACs with a density from 200 to 400 kg/m3 were tested. For the first time, commercial gypsum- and phosphogypsum-based hemp shive BAC fire performance was evaluated using a cone calorimeter. Results indicate that the role of gypsum content has a significant effect on the fire resistance. Time on ignition increased from 14 to 19 s and the heat release rate peak was reduced by 57%. Phosphogypsum binder, compared with commercial gypsum, showed a slight improvement of fire resistance as impurities with high water attraction are in the structure of PG.