In vitro reconstitution of protein systems--e.g., metabolic pathways, genetic circuits or biosensors--often requires optimization to enhance their activity. Combinatorial DNA libraries that simultaneously target multiple genes allow for a holistic optimization strategy by studying the interplay between the systems' components, which may reveal DNA variants that would be hidden when testing each element in isolation. Here, we screen large populations of synthetic vesicles that express combinatorial DNA variants of a DNA self-replicator or a phospholipid synthesis pathway. We simultaneously vary the strengths of multiple RBSs or synonymously mutate the first codons of multiple genes to explore the effects of the protein translation rates directly on the functionality of the two core synthetic cell modules. We isolated high performers through DNA self-selection or functional screening by fluorescence-activated cell sorting. Long-read sequencing of the fittest variants informed on the optimal RBS strengths and base substitutions in the first codons and indicated which genes were most impactful in regulating the functionality of the protein systems. Single-mutation data were used to predict the fitness of combinatorial variants, which was compared with the experimental fitness observed. The theoretical fitness of combinatorial variants was extremely predictive for the two-gene library of the DNA replicator but less for the larger pathway library. Altogether, our approach exemplifies how combinatorial testing can be expanded from single proteins to multiprotein systems, which can in the future be extended to the evolutionary engineering of even larger genetic and metabolic networks, and eventually an entire artificial cell.