Study of AMG 509 in Participants With Metastatic Castration-Resistant Prostate Cancer

Inclusion Criteria: * Parts 1, 2, 5 and 7: Participants with histologically or cytologically confirmed metastatic castration-resistant prostate cancer (mCRPC) who are refractory to a novel antiandrogen therapy (abiraterone acetate and/or enzalutamide, apalutamide, or darolutamide) and have failed at least 1 (but not more than 2) taxane regimens including for metastatic hormone-sensitive prostate cancer (mHSPC) (or who are deemed medically unsuitable to be treated with a taxane regimen or have actively refused treatment with a taxane regimen). Note: A taxane regimen is defined as a minimum exposure of 2 cycles of a taxane. Any NHT that has been administered and has been stopped for reasons other than progression will not be counted as an additional line of treatment. 1. Dose exploration phase: Novel antiandrogen therapy must have been given for treatment of metastatic disease. 2. Dose-expansion phase: participants must not have had more than 2 NHTs and 2 taxane regimens in any setting, and an additional up to 2 other systemic anti-cancer treatments are allowed (eg, anti-PD1, PARP inhibitors, radioligand therapies, sipuleucel-T, experimental agents) Note: Combinations are considered one systemic anti-cancer treatment. * Part 3: Participants with histologically or cytologically confirmed mCRPC who have received no or 1-2 prior NHTs (abiraterone acetate, enzalutamide, apalutamide, or darolutamide) given in any disease setting and who are deemed medically unsuitable to be treated with a taxane regimen or have actively refused treatment with a taxane regimen (unless taxane treatment was administered in HSPC setting). 0 1 prior PARP inhibitors or sipuleucel-T treatments are acceptable. Participants who received prior investigational therapy for the treatment of metastatic disease are not eligible. * Parts 4A, 4B and 7: 1. Participants with histologically or cytologically confirmed mCRPC who have received no or 1-2 prior NHTs (given in any disease setting depending on the part), and no or 1 taxane regimen (for HSPC). 2. Dose-expansion phase: at least 1 prior NHT must have been given; 0-1 prior PARP inhibitors are acceptable. 3. 4A: Participants planning to receive abiraterone acetate for the first time (participants who received prior abiraterone acetate are not eligible). Participants may have had exposure to up to 2 NHTs with a similar mechanism of action (apalutamide, enzalutamide or darolutamide) in the non-mCRPC and mCRPC setting. * Dose-expansion phase: up to approximately 10 participants with prior exposure to abiraterone acetate may be enrolled into Part 4A expansion cohort. d. 4B: Participants planning to receive enzalutamide for the first time (participants who received prior enzalutamide/apalutamide or daralutamide are not eligible). * Part 6: 1. Prior disease progression on 1, and only 1, NHT (either enzalutamide, apalutamide, or darolutamide) is required. NOTE: Prior progression on or intolerance to abiraterone is not allowed. 2. No prior treatment with any chemotherapy regimen in the mCRPC setting; ≤ 6 cycles of docetaxel treatment in the mHSPC setting is allowed. 3. mCRPC with ≥ 1 RECIST v1.1 measurable lesion that is present on baseline computed tomography (CT) or magnetic resonance imaging (MRI). * All parts: * Participants must have undergone bilateral orchiectomy or be on continuous androgen-deprivation therapy with a gonadotropin releasing hormone (GnRH) agonist or antagonist. * Total serum testosterone ≤ 50 ng/dL or 1.7 nmol/L. * Evidence of progressive disease, defined as 1 or more Prostate Cancer Working Group 3 (PCWG3) criteria: 1. PSA level ≥ 1 ng/mL that has increased on at least 2 successive occasions at least 1 week apart. 2. Nodal or visceral progression as defined by RECIST v1.1 with PCGW3 modifications. 3. Appearance of 2 or more new lesions in bone scan. * Eastern Cooperative Oncology Group performance status of 0-1. * Life expectancy ≥ 3 months. * Adequate organ function, defined as follows: 1. Hematological function: 1. absolute neutrophil count ≥ 1 x 10\^9/L (without growth factor support within 7 days from screening assessment). 2. platelet count ≥ 75 x 10\^9/L (without platelet transfusion within 7 days from screening assessment). 3. hemoglobin ≥ 9 g/dL (90 g/L) (without blood transfusion within 7 days from screening assessment). 2. Renal function: 1\. estimated glomerular filtration rate based on Modification of Diet in Renal Disease calculation ≥ 30 ml/min/1.73 m\^2. 3. Hepatic function: 1. aspartate aminotransferase (AST) and alanine aminotransferase (ALT) \< 3 x upper limit of normal (ULN) (or \< 5 x ULN for participants with liver involvement). 2. total bilirubin (TBL) \< 1.5 x ULN (or \< 2 x ULN for participants with liver metastases). 4. Cardiac function: 1. left ventricular ejection fraction \> 50% (2-D transthoracic echocardiogram \[ECHO\] is the preferred method of evaluation; multi-gated acquisition scan is acceptable if ECHO is not available). 2. Baseline electrocardiogram (ECG) QTcF ≤ 470 msec (average of triplicate values). 5. Pulmonary function: 1. baseline oxygen saturation \> 92% on room air at rest and no oxygen supplementation. Part 3-Retreatment group: * Deriving benefit from initial treatment with AMG 509 as evidenced by one of the following: 1. confirmed PSA50 response. 2. radiographic stable disease/partial response/complete response during 6 cycles of initial treatment with AMG 509 and without progression during the first 6 cycles. * No discontinuation for toxicity during the initial treatment with 6 cycles of AMG 509. * Progressive disease as defined in I106 within 12 months of final dose in their initial treatment with 6 cycles (EOT\_1). * Willingness to have a fresh tumor biopsy prior to initiating the additional course of treatment, depending on safety and feasibility as assessed by investigator. Key Exclusion Criteria: * Pathological finding consistent with pure small cell, neuroendocrine carcinoma of the prostate or any other histology different from adenocarcinoma. * Radiation therapy within 4 weeks of first dose (or local or focal radiotherapy within 2 weeks of first dose). * Untreated central nervous system (CNS) metastases or leptomeningeal disease. Participants with a history of treated CNS metastases are eligible if there is radiographic evidence of improvement upon the completion of CNS-directed therapy and no evidence of interim progression between the completion of CNS-directed therapy and the screening radiographic study. * Prior major surgery within 4 weeks of first dose. * Participants with symptoms and/or clinical signs and/or radiographic signs that indicate an acute and/or uncontrolled active systemic infection within 7 days prior to the first dose of investigational product administration. Simple urinary tract infections and uncomplicated bacterial pharyngitis are permitted if responding to active treatment and after consultation with sponsor. Screening for chronic infectious conditions is not required. * Confirmed history or current autoimmune disease or other diseases resulting in permanent immunosuppression or requiring permanent immunosuppressive therapy. * History of arterial or venous thrombosis (eg, stroke, transient ischemic attack, pulmonary embolism, or deep vein thrombosis); for arterial thrombosis within 12 months of AMG 509 initiation; for venous thrombosis, 6 months and stable on anti-coagulation. Participants with a recent history of venous thrombosis must be maintained on the same anti-coagulation therapy for a minimum of 28 days prior to first dose of study treatment. * Myocardial infarction and/or symptomatic congestive heart failure (New York Heart Association \> class II) within 12 months of first dose of AMG 509 with the exception of ischemia or non-ST segment elevation myocardial infarction controlled with stent placement and confirmed by a cardiologist more than 6 months prior to first dose of AMG 509. * Any anti-cancer therapy or immunotherapy within 4 weeks of start of first dose, not including luteinizing hormone-releasing hormone (LHRH)/GnRH analogue (agonist/antagonist). * Prior prostate specific membrane antigen (PSMA) radionuclide therapy within 2 months prior to AMG 509 unless participant received \< 2 cycles (Note: a participant cannot have received PSMA radionuclide therapy \< 35 days prior to enrollment if 1 cycle was given). Parts 3 and 4: prior PSMA radionuclide therapy is prohibited. Participants on a stable bisphosphonate or denosumab regimen for ≥ 30 days prior to enrollment are eligible (exception: part 3 retreatment). * Part 3-Retreatment only: Any anti-cancer therapy or immunotherapy, not including luteinizing hormone-releasing hormone/gonadotropin releasing hormone (LHRH/GnRH) analogue (agonist/antagonist), and/or bisphosphonate or denosumab regimen after last dose of AMG 509 initial course of treatment.