The term of office of each director is four years from the time of his appointment, except that for the board first appointed, the director appointed by the board as herein provided shall hold office for two years, and the remaining directors first appointed to the board shall classify themselves by lot, so that the two appointed by the board of supervisors shall hold office for one and three years and the two appointed by the city council shall hold office for two and four years.